ee . 
DOCUMENT RESUME ey 


~ 
| _— 


ED 163 4810 CS 008 516 
AUTHOR Deem, James 4M. 
TITLE - Needed: Systematic Evaluation of College Remedial 
- Reading Programs. 
PUB DATE Oct 78 : 
NOTE 14p.; Paper presented at the Annual Meeting of the 
North Central Reading Association (21st, Notre Dame, 
. Indiana, October 20-21, 1978) 
EDRS PRICE MF-$0.83 HC-$1.67 Plus Postage. 
DESCRIPTORS Basic Skills; *College Curriculum; *Curriculua 


> : Evaluation; Educational Needs; Evaluation Methods; 
*Evaluation Needs; Higher Education; Literature 
Reviews; Models; *Program Evaluation; *Program 
Improvement; *Remedial Reading Programs : 


ABSTRACT 

. Although, as an important component of any basic 
skills curriculum, college remedial reading programs need to be 
objectively and fully evaluated, no coaprehgnsive, systematic 
procedures have been-developed for this purpose. Three guidelines 
should be followed in.evaluating college remedial reading programs: 
first, appropriate variables that allow for maximum input must be 
selected; second, the evaluation must cover not only the performance 
of the students in the program but the entire program itself; and, 
finally, the end result of the evaluation aust be prograa 
improvement. Since no complete evaluation model has been developed to 
aid college remedial reading program directors, directors must rely . 
on the few evaluation models that have been proposed. Until aie: 
meaningful systematic evaluations of college remedial reading \ 
programs can be carried out, the success of such programs and their ‘ / 
various methodologies--in short, the need for such programs--will be 
seriously questioned. (The paper includes an outline of one model for 
program evaluation and a bibliography of relevant readings.) (GW) 
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NEEDED: SYSTEMATIC EVALUATION, 
OF COLLEGE REMEDIAL READING PROGRAMS* 


\ 
Are college remedial reading (CRR) programs failing to help © i 
under-prepared high-risk students? Would these students do just as | 
wel] SrEiRUe parbicipattne in such remedial programs? 
With the influx of high-risk students into institutions of 
higher education, greater demands have been placed on the college 
basic skills programs "to be more accountable than traditional 
education programs" (Grant and Hoeber, 1978, p. 34). Basic skills 
program directors, however, have often been reluctant ‘to conduct 


objective or detailed evaluations to determine the effects of a 


program, since a negative result may mean the elimination of such 


_-programs. Indeed, "there is strong evidence that many college 


administrators and state legislators look at evaluation as a means 
to legitimate the curtailment of such programs" (Bynum et al, 1972, 
do. 82). As a result of these problems, few evaluations are published; 
even these are often "Timi ted in quantity and scope" (Bynum, p. 120) and 
do not address the issue of program modification, since ‘change is seen 
as a threat to program continuation. Even those directors who may wish: 
to evaluate their programs objectively cannot find an adequate evaluation 
model Consequently, every basic skills program that is evaluated uses 
different--and often, inadequate--measures of effectiveness. 

College remedial reading programs especially, as an important 


e 


component of any basic skills curriculum, need to he objectively and 
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: any evaluated. As Chaplin (1977) notes, during the 1960's such 


‘reading programs “were hurriedly instituted with little foresight 

and planning for their stability) They were simply insurance 
policies that protected the cherished high standards of academic 
excel Tence" (ps2). consenuentiy.\ ieee has been done to bring 
about their evaluation ina systematic or realistic way. Further, 
br the existing evaluations, only a few have been published 
(Roueche and Kirk, 1973) or made avaflable for study in the ERIC 
system \Losak, 1971; Schiavone, 1977). 

College reading educators have wrestled with the issue of 
evaluation for the past twenty-five uERR Ss Eller (1956) and Colvin 
(1962) each called for the evaluation of college reading programs : 
but did not specify any procedures for such\ evaluations. Summerfeld 7 
(1955), however, did propose five general steps for conducting 
evaluations of college reading programs. Nonetheless , he, as well as 
other educators of that tine, based his evaluation procedures on a 
research model, in contrast to the current evaluation model. 

Avadty ional tys the most common method of evalyating the - 
effectiveness of coflege remedial reading programs ns been based on 
the testing of one Jivinbte: usually the improved nih achievement 
or reading scores of program participants. However, as \Guba (1972) 
states, "the use of laboratory research designs and techniques poses 
Sond tions that are simply inappropriate for the purposes for which 
one does an evaluation" (p. 257-258). First, such research studies 


rely upon randomly-selected experimental and control groups to Produce 
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statistically valid results. However, control groups and random 
selection are inconsistent with the philosophies of CRR p shan 

and the needs of participating high-risk students. ate 
research study cannot reflect the effects of an entire Py gran by 
the use of one or two variables, merely a controled portion af 1£< 
On the other hand, evaluations are done by setting up "condi tions of 
invited interference from all factors that might ever influence a 
learning...transaction" (Guba, p. 257). The goal is less, not \ 
more, control. Finally, research is rarely undertaken to produce 
change, merely to “search for new knowledge regardless of [its] 
value" (Suchman, 1967, p. 75). Evaluation sex imnae:, paceesart ly: the 
need for change. 

There are no solutions currently to the problems of evaluating 
CRR programs for high-risk students. However, my research in the 
field has indicated that three guidelines are necessary in conducting 
any such evaluation. First, appropriate variables which allow for 
maximum input must be selected. Second, the evaluation must cover not 
only the performance of the students in the program, but the entire 
program itself. Finally, the end result of the evaluation must be 
program improvement. 

First, one must select comprehensive variables in conducting an 
evaluation of CRR programs. However, there is little in the existing 
literature to guide the CRR program evaluator. Maxwell (1971) proposed 
four vartables in evaluating CRR programs: GPA, the limited use of stan- 


dardized reading tests, local objectives, and attitude changes. Drexler 
ff 
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and Pepper (1972) clearly favored the use of academic achievement but 
conceded that attrition, persistence, or "the work a student: anaxdes 
in after graduation" might also be used as measures of program effect- 
iveness (p. 190). Emond (1976) suggested four longitudinal measures : 
persistence, semester.-by-semester GPA, number of failures and course 
withdrawals, and attitude changes. Emond felt that persistence 
especially "has been overlooked as a major indicator of program success" 
(p. 23). Predgier (1965) wrote that persistence should receive more . 
attention as a criterion for success, for "ultimately, the student's 
success in college is judged not in terms of his GPA, but, rather, in 
terms of the educational program which he has completed" (p. 62). 
Whether any or all of these yariables are appropriate for programs 
involving high-risk students, or whether all of the necessary variables 
have been identified, one cannot say. . 
Second, one must evaluate the entire program. In the past, 
_ program evaluation has too often been confused with student evaluation 
(Bynum). Although one can evaluate certain aspects of a program by: 
evaluating the students and their performance, problems can result from 
such an emphasis. If the student does not succeed in the program, 
oftentimes "the student is viewed as the failure rather than the 
orogram" (Grant and Hoeber, p. 31). Moreover, little attention is 
paid to the processes or methods of the program itself. For ane ed 
Schiavone (1977) judged the effectiveness of a remedial reading program 
at Manhattan Community College by studying the student pass-rate for the 
program over six consecutive semesters. Schiavone found that the pass- 


rate declined thirteen percent over the six semesters from 68 to 55 
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percent. Although one could take issue with the standard set for 


passing the course (which was the attainment of the 12th grade reading 
level as measured by a raw seare of 50 on the Stanford Test of Academic 
Skills [TASK]), the main problem is that the evaluation implied that ; 
the students, not the srauren, "have failed. Schiavone recommended the 
expansion of the course into a "sequence of levels"(p. 34); in short, P 
a lengthening of the current program. By focusing on the students and 
their declining performance, Schiavone ignored any implications of 
prgeagial allure. Deciding why the pass-rate declined 13 percent in six 
semesters would have been the first step in a program-centered evaluation. 
. Finally, one must aim for the improvement of the program not 
the maintenance of the status quo in conducting CRR program evaluations. 
Two questions should be answered: what aspects of the program are not 
working as well as they could be? Then, what changes can be made so 
that the program can effectively meet more of its objectives? While 
this point may seem obvious, this has oftentimes not been the case in 
existing evaluations. As example of this problem surfaces in a study 
by Losak (1971), in which a random sample of freshmen students at the 
Miami-Dade Junior College who scored low on a placement test were 
enrolled in a regular freshman English course rather than the remedial 
program. The experimental group received instruction in remedial 
reading-writing, while the control group received instruction in the 
regular freshman English course. Losak found that the current remedial 
program was no more effective for the remedial students than the English 


course was for the students in the control group. More importantly, he 
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concluded that there was a need to identify those who could and could 
not benefit from a standard remedial program; those who could nog, 
benefit (i.e., those who failed) should be enrolled in more "relevant 
programs [that were] developed for them" (p. 13). In other words, he 
suggested that the students be tailored to the program. If the program 
does not succeed, one must find the students who will make it succeed 
rather than change the) program. 

What is needed now is the framework of a complete evaluation 
model to aid CRR program directors. Because none have been suggested, 
one must rely on the few models proposed for basic skills programs. 

Bynum (1972), for example, in a comprehensive study of 711 
college compensatory programs, presented a sample evaluation from one 
anonymous state university in New York. This evaluation reported 
student performance data as well as "the relative impact of specific 
practices and the contribution of specific courses" (p. 81). However, 
the evaluation procedure is not presented in enough detail to warrant 
‘its use as a model. Grant and Hoeber (1978) suggested two models for 
the evaluation of basic skills programs. One, designed by the Fund for 
the Improvement of Postsecondary Education, “proposed a three part 
evaluation method" (p. 36). While this model may prove useful when it 
is disseminated, it is currently unavailable from a publisher or from the 
ERIC system, and not readily available from FIPSE. The second model 
alluded to by Grant and Hoeber was proposed by Suchman (1967) to be used 
in evaluating public service and social action programs (Tables 1 & 2). 


In following the model, one would first establish long-range, intermediate, 
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and immediate objectives, as well as the assumptions implicit in the 
objectives. Then one would evaluate the program in terms of the 
objectives by using a aye category evaluation procedure. What is 
interesting about Suchman's model is that it employs both evaluative 
and research designs within its five categories. Furthermore, the 


categories (Effort, Performance, Adequacy of Performance, Efficiency, 


and Process) are arranged hierarchically from the least to the most 
difficult to accomplish. Such a ‘framework may prove useful for the 
evaluation of CRR programs. 

Until more meaningful systematic evaluations of CRR programs 
can be carried out, the success of such programs and their various 
methodologies--in short, the need for such programs --will be seriously 
questioned. Further explorations to find appropriate comprehensive and 
systematic evaluation models for CRR programs must be conducted, 
perhaps experimenting with Suchman's model, and the results published 
or stored ja. ‘the ERIC system. Without such information, the worth, 


as well as the future, of such programs is in doubt. 


+. 


TABLE 1 ( 


Suchman's Model of Program Evaluation * 


I. Set Objectives 
(long-range, intermediate, immediate) 


| II. Determine Underlying Assumptions 
III. Carry Out Evaluation Using Five Categories 
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* A B 
Effort Performance Adequacy of Efficiency 
Performance 


Type of Criterion | Evaluative Administrative Administrative 
Related Question How much effort How much was How sufficient Could the same 
_was expended? accomp1 ished? were these accomp]ishments 
accomplishments? haye been produced 
more efficiently? 
Evaluative Goals amount of activity | accomplishments accomplishments need for alternate 
; in terms of in terms of total methods in terms 
objectives need of cost, time, 
personnel 


No judgment of poorly-stated or unrealistic. subjective basis 
effort missing objectives | expectations; for judgment 
subjective 
judgment 


Evaluative 
Problems 


# students meet- Attendance ratios; | Are students who 

ing objectives; drop ratios; pro- need the program 

# competencies portion of popula- | being identified 

met; # students tion in program during diagnosis? © 

passing, etc. ‘vs. total diagnosed] Would a different 
population. | approach produce 

the same results? 


Possible 
application for 
individualized 


Hours of oper- 
ation; # faculty, 

students, classes; 
# units of faculty 
material produced; 
# work assignments 
finished daily, 


weekly, hourly. 


reading program in 
laboratory setting 


E 
Process 


Research 


What are the 
successful and 
unsuccessful 
components of the 
UrOd ail 
identification of 
successful and 
unsuccessful 
aspects 


unreliable and 
invalid measures 


SEE REVERSE SIDE 
ia 


Adapted from Evaluative Research: Principles and Practice im Public Servi 5 
and Social Action Programs. by Edward A. Suchnan. New York: RusseTi sage fin., 1967. 11] 


TABLE 2 


Subcategories of the Process Category: 
One of Suchman's Five Categories of Evaluation * 


Subcategories of Process 


Possible Application for Individualized 
Reading Program in Laboratory Setting 


A. Attributes of the program that 
make 1t successful or: 
unsuccessful 


B. Recipients of the program who 
are affected; successful and. 
unsuccessful recipients 


C. Conditions under which the 
program 1s successful 


D. Effects produced by the 
program (single/multiple; 
intentional/unintentional ; 
duration): 


1. Cognitive Effects 
2. Attitudinal Effects 


3. Behavioral Effects 


oes 


Individualization? 
Competency-based? 
Whole-class approach? 
Interpersonal contact? 
Relevant Material? 


Students classified by their progress 
toward objectives (SES, age, race, no. 
of semesters in attendance, etc.) 


rs 


varying class sizes? ; 
mechanical vs. nonmechanical devices? 
lab vs. class? 


measured by GPA, reading scores, etc. 


measured by self-concept and attitude 
instruments, etc. 


measured by attrition, persistence, etc. 


o 


i . 

. Adapted from Evaluative Research: Principles and Practice in Public 
Service and Social Action Programs by Edward A Suchman. New York: Russell 
Sage Foundation, 196/. 
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